Systems and methods for training machine learning models

ABSTRACT

Methods and computer-readable media for repeated holdout validation include collecting independent data representing independent variables; collecting dependent data representing a dependent variable; correlating the independent data with the dependent data; creating a data set comprising the correlated independent and dependent data; generating a plurality of unique seeds; creating a plurality of training sets and a plurality of validation sets; associating each training set with a single validation set; training the neural network a plurality of times with the training sets and seeds to create a plurality of models; calculating accuracy metric values for the models using the validation sets associated with the training sets used to create respective models; performing a statistical analysis of the accuracy metric values; and ranking the independent variables by a strength of correlation of individual independent variables with the dependent variable, when a metric of the statistical analysis exceeds a threshold.

TECHNICAL FIELD

The present disclosure generally relates to methods andcomputer-readable media for evaluating survey datasets using machinelearning models.

BACKGROUND

Many applications, such as financial systems, supply line management,transportation, and industrial processes, use machine learning models todiscover relationships between variables relevant to the application.However, large training datasets may be required to train complexmachine learning models including many model parameters.

Numerical methods, such as repeated holdout sampling, can be used tocompensate for small datasets. In repeated holdout sampling, subsets ofthe training dataset are repeatedly selected and used to train themachine learning model, while using the remainder of the dataset formodel validation. However, machine learning models trained usingrepeated holdout sampling can exhibit sensitivity to small changes intraining, such as the particular sequence of training data presented orthe particular division of the training dataset into training andvalidation subsets. Accordingly, machine learning models generated fromthe same training dataset using repeated holdout sampling may providesubstantially different outputs for the same input data. Suchdifferences may be especially pronounced for complex machine learningmodels trained on smaller datasets.

SUMMARY

The disclosed embodiments include a computer-implemented method. Themethod includes the steps of collecting independent data representingindependent variables; collecting dependent data representing adependent variable; correlating the independent data with the dependentdata; creating a data set comprising the correlated independent anddependent data; generating a plurality of unique seeds; creating aplurality of training sets and a plurality of validation sets;associating each training set with a single validation set; training aneural network a plurality of times with the training sets and seeds tocreate a plurality of models; calculating accuracy metric values for themodels using the validation sets associated with the training sets usedto create respective models; performing a statistical analysis of theaccuracy metric values; and ranking the independent variables by astrength of correlation of individual independent variables with thedependent variable, when a metric of the statistical analysis exceeds athreshold.

The disclosed embodiments further include a non-transitorycomputer-readable medium. This computer-readable medium containsinstructions to perform operations including collecting independent datarepresenting independent variables; collecting dependent datarepresenting a dependent variable; correlating the independent data withthe dependent data; creating a data set comprising the correlatedindependent and dependent data; generating a plurality of unique seeds;creating a plurality of training sets and a plurality of validationsets; associating each training set with a single validation set;training a neural network a plurality of times with the training setsand seeds to create a plurality of models; calculating accuracy metricvalues for the models using the validation sets associated with thetraining sets used to create respective models; performing a statisticalanalysis of the accuracy metric values; and ranking the independentvariables by a strength of correlation of individual independentvariables with the dependent variable, when a metric of the statisticalanalysis exceeds a threshold.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, illustrate disclosed embodiments and,together with the description, serve to explain the disclosedembodiments. In the drawings:

FIG. 1 depicts a flowchart of an exemplary method for training a machinelearning model, consistent with the disclosed embodiments.

FIG. 2 depicts an exemplary system for implementing a machine learningmodel, consistent with disclosed embodiments.

FIG. 3 depicts an exemplary system for generating a survey dataset,consistent with disclosed embodiments.

FIG. 4 depicts exemplary training of machine learning models, consistentwith disclosed embodiments.

FIG. 5 depicts an exemplary method for identifying predictive machinelearning parameters using multiple machine learning models, consistentwith disclosed embodiments.

DESCRIPTION

In the following detailed description, numerous specific details are setforth in order to provide a thorough understanding of the disclosedexample embodiments. However, it will be understood by those skilled inthe art that the principles of the example embodiments may be practicedwithout every specific detail disclosed herein. Well-known methods,procedures, and components have not been described in detail so as notto obscure the principles of the example embodiments. Unless explicitlystated, the example methods and processes described herein are notconstrained to a particular order or sequence, or constrained to aparticular system configuration. Additionally, some of the describedembodiments or elements thereof can occur or be performedsimultaneously, at the same point in time, or concurrently.

Reference will now be made in detail to the disclosed embodiments,examples of which are illustrated in the accompanying drawings.

As described above, machine learning models trained on small datasetscan exhibit high variability (e.g., significant variation between modelsin outputs generated given the same or similar inputs). In particular,machine learning models trained to predict behavior from survey data canexhibit high variability, as surveys typically include many questionsand often have low response rates.

The disclosed embodiments address the problem of machine learning modelvariability by training multiple models, identifying models satisfying aprediction criterion, and identifying important model parameters for theidentified models. In some embodiments, the important model parameterscan be identified based on the contributions of the model parameters toeach of the identified models. For example, the k-most important modelparameters in each of the identified models can be identified. Invarious embodiments, the important model parameters can be identifiedbased on a function of the contributions of the model parameters to eachof the identified models. For example, the contributions of the modelparameters to at least some of the identified models can be combined(e.g., averaged or weighted averaged). The k-most important modelparameters, so combined, can be identified. The identified parameterscan be used to inform actions concerning the model outputs (e.g., topredict the best way to modify undesirable behavior based using limitedsurvey data). The identified parameters, or functions of the identifiedparameters, can be used as inputs for additional machine learning (e.g.,model parameters identified as relevant using a regression analysis ordecision tree analysis can be used as features for a neural networkmodel).

Consistent with disclosed embodiments, datasets may include elementscorresponding to real-world conditions, events, and/or measurements. Theelements may include variables having associated values. The variablesmay include independent and dependent variables, where independentvariables correspond to actual or potential model parameters anddependent variables correspond to actual or potential model outputs. Thedatatypes of the values are not intended to be limiting. For example, adataset may include transaction data, financial data, demographic data,public data, government data, environmental data, traffic data, networkdata, transcripts of video data, and/or other data. A dataset may be ina variety of data formats including, but not limited to, PARQUET, AVRO,SQLITE, POSTGRESQL, MYSQL, ORACLE, HADOOP, CSV, JSON, PDF, JPG, BMP,and/or other data formats.

Consistent with disclosed embodiments, a machine learning model mayinclude a decision tree model, regression model (e.g., linear orlogistic regression model), support vector machine, k-nearest neighbormodel, neural network, or the like. Given a set of input data, themachine learning model can generate a label or classification for theinput data, predict a corresponding set of output data, or perform othertasks. The machine learning model can be trained using a set of trainingdata. Depending on the type of machine learning model, this training canbe supervised, semi-supervised, or unsupervised. The machine learningmodel can be implemented using one or more computing devices (e.g.,wearable device, smartphone, laptop, desktop, workstation, server, cloudcomputing platform, or the like). The one or more computing devices canbe configured to obtain input data. For example, the one or morecomputing devices can generate the input data (e.g., by takingmeasurements), receive the input data (e.g., from other devices), and/orretrieve the input data (e.g., from one or more remote or localdatabases).

FIG. 1 depicts exemplary computer-implemented process 100 for training amachine learning model, consistent with disclosed embodiments. Thetraining process can be performed using one or more computing devices.The one or more computing devices can be configured with instructions(e.g., computer code) to perform the training. While the same computingdevices are described as performing each step of process 100, forsimplicity of description, different computing devices or groups ofcomputing devices can be used to perform different steps (or differentfunctions within each step). Consistent with disclosed embodiments,process 100 can include steps of obtaining a dataset, training multiplemachine learning models, and identifying important parameters of theresulting collection of machine learning models.

In step 102 of process 100, the one or more computing devices can obtaindata, consistent with disclosed embodiments. For example, the data canbe generated by the one or more computing devices, received from one ormore other systems, and/or retrieved from one or more other systems. Insome embodiments, values for at least one independent variable may beobtained separately from values for at least one dependent variable. Forexample, values for independent variables may be generated from businessrecords, while values for dependent variables may be generated fromcustomer survey results. As a further example, when values forindependent variables are generated from business records, suchvariables may include business measurements such as sales value, salesquantity, sales frequency, transaction time, user referrals, and/or asuccess ratio of completed transactions to initiated transactions. Tocontinue this example, when values for dependent variables are generatedfrom customer survey results, such values can be the results asobtained, or can reflect additional analysis or processing of theresults (e.g., net values calculated from the obtained survey results).

Returning to FIG. 1 , in step 104 of process 100, the one or morecomputing devices can associate values of independent variables withvalues of dependent variables, consistent with disclosed embodiments.That is, individual values of dependent variable can be matched toindividual values of independent variables. This association can dependon a predetermined relationship between the matched dependent variablevalues and independent variable values. As a non-limiting example, thedependent variable values and independent variable values may be fromthe same source (e.g., same sensor, business establishment, individual,etc.). For example, if the independent data are measurements of windspeed, angle of attack, air density, air temperature, and ambientpressure, and the dependent data is a measurement of force generated bya wing, individual measurements may be correlated based on the time atwhich measurements were taken. As another example, if the independentdata are measurements of sales value, transaction success rate, andsales frequency, and the dependent variable is a user satisfactionmetric obtained by a survey answered by a user, individual data pointsmay be correlated if they originate from the same user. In someembodiments, there may be more independent data points than dependentdata points. This could occur, for instance, if a user's transactionhistory is available but the user did not answer a survey. In this case,the surplus data may be discarded at step 104.

In step 106 of process 100, the one or more computing devices can createan original training dataset comprising the associated independent anddependent data, consistent with disclosed embodiments. A non-limitingexample of creating such a dataset is detailed below with regards toFIGS. 4 and 5 . The dataset may include elements (e.g., rows, objects,documents, etc.), each element containing associated values of theindependent and dependent variables. The original training dataset maybe stored in a database for future analysis or may be immediatelyanalyzed.

The one or more computing devices can then generate a plurality ofmachine learning models. In some aspects, such generation can includeobtaining pairs of training and validation datasets (e.g., creating,retrieving, or receiving such pairs). For example, creation of eachassociated training and validation dataset can include selection ofelements in the original training dataset and assignment of the selectedelement to either the training or validation dataset. The particularmethod for assigning elements to training or validation datasets is notintended to be limiting. For example, the elements can be selectedrandomly, pseudorandomly, or deterministically.

In some embodiments, as an additional example, in step 108 of process100, the one or more computing devices can create a plurality of uniqueseeds. The unique seeds can be used to create each machine learningmodel. For example, one or more of the unique seeds can be used toinitialize a random number generator used to perform tasks related togeneration of a machine learning model (e.g., creation of the trainingand validation datasets, performance of training updates, etc.). Theseseeds may be generated randomly, by a random number generator, orpseudo-randomly. The seeds may be, for instance, an individual number,or an array of multiple numbers. The seeds may be stored in a databaseto replicate past results of training the neural network.

In some embodiments, as an additional example, in step 110 of process100, the one or more computing devices can create the plurality oftraining and validation datasets. In some instances, the training setsand validation sets are subsets of the original training dataset createdat step 106, and include elements of this dataset (e.g., sets of valuesfor the independent and dependent variables). In some embodiments, thetraining sets and validation sets can together include all, orsubstantially all (e.g., >90%), of the elements in the original trainingdataset. For example, 75% of the data set may be assigned to a trainingset, and 25% of the data set may be assigned to a correspondingvalidation set. Assigning an individual element of the original datasetto a training set or a validation set may be based on a random number,or a pseudo-random number. For instance, in a simplified example, a dataset may contain 30 data points, each containing independent data pointsand a dependent data point, and process 100 may assign 20 data points toa training set and 10 data points to a validation set. Process 100 maygenerate 10 random integers between 1 and 30. The 10 random integers maycorrespond to 10 data points, which are assigned to the validation set.The remaining, unchosen data points may be assigned to the training set.This may then be repeated multiple times to generate many training setsand validation sets.

In some embodiments, as an additional example, in step 112 of process100, the one or more computing devices can associate each training setwith a single validation set. Furthermore, in some embodiments, step 112includes pairing training and validation sets such that no individualdata point is in both a training set and a validation set of a pair.This ensures that the neural network is not validated with the same datawith which it is trained, which may lead to deceptively high modelaccuracy. For example, a data set may contain individual data points A,B, and C. A first training set may include points A and B, and a secondtraining set may include points B and C. A first validation set, pairedwith the first training set, would then include data point C, and wouldnot include data points A or B so there is no intersection between thetraining set and validation set. Similarly, a second validation set,paired with the second training set, would include data point A, but notdata points B or C. Furthermore, this example illustrates that anindividual data point may alternately be in a training set and avalidation set of different pairs, but is not in a training set and avalidation set of the same pair.

In some embodiments, the union of a paired training set and validationset may be equal to the original data set. In this way, every data pointis included in a pair, either as part of the training set or thevalidation set. Alternatively, some data points may be excluded from atraining set/validation set pair. This may occur, for instance, if thereare millions of data points and computational limits only permitprocessing of a hundred thousand data points simultaneously.

In some embodiments, as an additional example, in step 114 of process100, the one or more computing devices can train the machine learningmodels using the training and validation datasets, consistent withdisclosed embodiments. For example, when the machine learning modelincludes one or more decision trees, the one or more computing devicescan train the machine learning models using the XGBoost softwarelibrary, or a similar tool. In this case, a model may be created byoperating XGBoost with inputs of a seed value and a paired training setand validation set. In this way, different models may be created formany different seed and training set combinations. In some embodiments,a number of training sets and a number of validation sets are equal to anumber of unique seeds. A number of models is also equal to the numberof unique seeds, and each model is created by training with one seed andone pair of a training set and a validation set. In this way, every seedand every pair of training set and validation set may be used to trainand evaluate a model, and no two models share the same seed, trainingset, or validation set, thereby creating greater diversity in modeltraining and leading to models that more robustly analyze complex datasets. Though described above with regards to decision trees, one ofskill in the art would appreciate that the disclosed embodiments are notso limited.

In step 116 of process 100, the one or more computing devices cancalculate accuracy metric values for the models using the validationsets associated with the training sets used to create respective models,consistent with disclosed embodiments. For example, after a model iscreated using a training set, the model accuracy is evaluated using thevalidation set that was associated with the training set at step 112. Anaccuracy metric value may be calculated for each model, or a portion ofthe models. The accuracy metric value can reflect how accurately eachmodel predicts the validation data provided to it. Any accuracy metricmay be used in step 116, such as a true positive rate or a true negativerate. In some embodiments, the Area Under Curve (AUC), also known as theArea Under the Receiver Operating Characteristics (AUROC), may be used.In the non-limiting decision tree example provided above, XGBoost can beused to determine the AUC for each model. As would be appreciated bythose of skill in the art, other suitable training packages may be usedto calculate the AUC, or another suitable accuracy metric, for machinelearning models generated using such a training package.

In step 118 of process 100, the one or more computing devices can rankmodel parameters based on the contribution of the model parameters tothe machine learning models (e.g., a degree or sensitivity of dependenceof values of the dependent variables on the values of the independentvariable). For example, in the non-limiting decision tree exampleprovided above, XGBoost can be configured to provide metrics such asweight, gain, cover, and/or similar metrics describing the contributionof decision tree features to the predictive power of the decision treemodel. As would be appreciated by those of skill in the art, similarmetrics can be provided for other types of machine learning modelsand/or when using other training packages. In some embodiments, the rankof a model parameter can depend on the contribution of the modelparameter to a particular machine learning model, or to the models inthe plurality of machine learning models. For example, a degree of thecontribution may be the average contribution across the models in theplurality of machine learning models. For example, in a systemevaluating the prevalence of diabetes, independent variables may be age,sex, a fasting blood sugar level, and the dependent variable may be abinary indicator of diabetes or no diabetes. The one or more computingdevices can rank these model parameters by first calculating the averageweight associated with age in the models, the average weight associatedwith sex, and the average weight associated with fasting blood sugar.These averages may then be ordered. For instance, the average weight forage may be the greatest magnitude (i.e., largely positively ornegatively correlated). Such results may support a conclusion that agecontributed the most to the output classification of diabetes or nodiabetes. As described above, this conclusion may be more reliable thana conclusion based on a single model, as single models trained onlimited datasets exhibit increased variance.

In some embodiments, the one or more computing devices can determinewhether the plurality of machine learning models satisfies an accuracycriterion. For example, the one or more computing devices can determineone or more statistics (e.g., mean, median, standard deviation, or thelike) of the accuracy metric values for the plurality of machinelearning models. The one or more computing devices can be configured toend the analysis without ranking the model parameters, or not provide orreturn the ranked model parameters, when the one or more statisticsfails to satisfy the accuracy criterion. For example, the accuracycriterion may be a threshold value (e.g., a predetermined value between0.5 and 0.8) and the one or more computing devices can be configured toend the analysis without ranking the model parameters when an average ofthe accuracy metric values for the plurality of machine learning modelsdoes not exceed this threshold value. In some embodiments, the one ormore computing devices can perform the ranking using only the machinelearning models having accuracy metric values satisfying the accuracycriterion. For example, when two of three machine learning models haveaccuracy metric values satisfying the accuracy criterion (e.g.,exceeding a predetermined threshold), the one or more computing devicescan perform the ranking using only these two machine learning models.

FIG. 2 depicts an exemplary computing system (computing system 200)suitable for implementing process 100, consistent with disclosedembodiments. Computing system 200 may include a computing device, acomputer, a server, a server cluster, a plurality of clusters, and/or acloud service, consistent with disclosed embodiments. As shown,computing system 200 may include one or more processors 210, one or moreI/O devices 220, and one or more memory units 230. In some embodiments,some or all components of computing system 200 may be hosted on adevice, a computer, a server, a cluster of servers, or a cloud service.In some embodiments, computing system 200 may be a scalable systemconfigured to efficiently manage resources and enhance security byprovisioning computing resources in response to triggering events andterminating resources after completing a task (e.g., a scalable cloudservice that spins up and terminates container instances).

FIG. 2 depicts an exemplary configuration of computing system 200. Aswill be appreciated by one skilled in the art, the components andarrangement of components included in computing system 200 may vary. Forexample, as compared to the depiction in FIG. 2 , computing system 200may include a larger or smaller number of processors, I/O devices, ormemory units. In addition, computing system 200 may further includeother components or devices not depicted that perform or assist in theperformance of one or more processes consistent with the disclosedembodiments. The components and arrangements shown in FIG. 2 are notintended to limit the disclosed embodiments, as the components used toimplement the disclosed processes and features may vary.

Processor 210 may comprise known computing processors, including amicroprocessor. Processor 210 may constitute a single-core ormultiple-core processor that executes parallel processes simultaneously.For example, processor 210 may be a single-core processor configuredwith virtual processing technologies. In some embodiments, processor 210may use logical processors to simultaneously execute and controlmultiple processes. Processor 210 may implement virtual machinetechnologies, or other known technologies to provide the ability toexecute, control, run, manipulate, store, etc., multiple softwareprocesses, applications, programs, etc. In another embodiment, processor210 may include a multiple-core processor arrangement (e.g., dual core,quad core, etc.) configured to provide parallel processingfunctionalities to allow execution of multiple processes simultaneously.One of ordinary skill in the art would understand that other types ofprocessor arrangements could be implemented that provide for thecapabilities disclosed herein. The disclosed embodiments are not limitedto any type of processor. Processor 210 may execute various instructionsstored in memory 230 to perform various functions of the disclosedembodiments described in greater detail below. Processor 210 may beconfigured to execute functions written in one or more known programminglanguages.

I/O device 220 may include at least one of a display, an LED, a router,a touchscreen, a keyboard, a microphone, a speaker, a haptic device, acamera, a button, a dial, a switch, a knob, a transceiver, an inputdevice, an output device, or another I/O device to perform methods ofthe disclosed embodiments.

I/O device 220 may be configured to manage interactions betweencomputing system 200 and other systems using a network. In some aspects,I/O device 220 may be configured to publish data received from otherdatabases or systems not shown. This data may be published in apublication and subscription framework (e.g., using APACHE KAFKA),through a network socket, in response to queries from other systems, orusing other known methods. Data may be survey data or transaction data,as described herein. In various aspects, I/O 220 may be configured toprovide data or instructions received from other systems. For example,I/O 220 may be configured to receive instructions for generating datamodels (e.g., type of data model, data model parameters, training dataindicators, training parameters, or the like) from another system andprovide this information to programs 236. As an additional example, I/O220 may be configured to receive data including sensitive data fromanother system (e.g., in a file, a message in a publication andsubscription framework, a network socket, or the like) and provide thatdata to programs 236 or store that data in, for example, data set 232 ormodels 234.

In some embodiments, I/O 220 may include a user interface configured toreceive user inputs and provide data to a user (e.g., a data manager).For example, I/O 220 may include a display, a microphone, a speaker, akeyboard, a mouse, a track pad, a button, a dial, a knob, a printer, alight, an LED, a haptic feedback device, a touchscreen and/or otherinput or output devices.

Memory 230 may be a volatile or non-volatile, magnetic, semiconductor,optical, removable, non-removable, or other type of storage device ortangible (i.e., non-transitory) computer-readable medium, consistentwith disclosed embodiments. As shown, memory 230 may include data set232, including one of at least one of encrypted data or unencrypteddata. Memory 230 may also include models 234, including weights andparameters of neural network models.

Programs 236 may include one or more programs (e.g., modules, code,scripts, or functions) used to perform methods consistent with disclosedembodiments. Programs may include operating systems (not shown) thatperform known operating system functions when executed by one or moreprocessors. Disclosed embodiments may operate and function with computersystems running any type of operating system. Programs 236 may bewritten in one or more programming or scripting languages. One or moreof such software sections or modules of memory 230 may be integratedinto a computer system, non-transitory computer-readable media, orexisting communications software. Programs 236 may also be implementedor replicated as firmware or circuit logic.

Programs 236 may include a machine learning framework 238 (e.g., XGBOOSTor the like) and a statistical analysis program 240. Modules of programs236 may be configured to receive, retrieve, and/or generate models,consistent with disclosed embodiments. Modules of programs 236 may beconfigured to perform operations in coordination with one another. Insome embodiments, programs 236 may be configured to conduct anauthentication process, consistent with disclosed embodiments.

Machine learning framework 238 may include programs (scripts, functions,algorithms) to assist creation of, train, implement, store, receive,retrieve, and/or transmit one or more machine learning models. Machinelearning framework 238 may be configured to assist creation of, train,implement, store, receive, retrieve, and/or transmit, one or moreensemble models (e.g., models comprised of a plurality of models). Insome embodiments, training of a model may terminate when a trainingcriterion is satisfied. Training criteria may include number of epochs,training time, performance metric values (e.g., an estimate of accuracyin reproducing test data), or the like. Machine learning framework 238may be configured to adjust model parameters and/or hyperparametersduring training. For example, machine learning framework 238 may beconfigured to modify model parameters and/or hyperparameters (i.e.,hyperparameter tuning) using an optimization technique during training,consistent with disclosed embodiments. Hyperparameters may includetraining hyperparameters, which may affect how training of a modeloccurs, or architectural hyperparameters, which may affect the structureof a model. Optimization techniques used may include grid searches,random searches, gaussian processes, Bayesian processes, CovarianceMatrix Adaptation Evolution Strategy techniques (CMA-ES),derivative-based searches, stochastic hill-climbing, neighborhoodsearches, adaptive random searches, or the like.

In some embodiments, machine learning framework 238 may be configured togenerate models based on instructions received from another component ofcomputing system 200 and/or a computing component outside computingsystem 200. For example, machine learning framework 238 can beconfigured to receive a visual (e.g., graphical) depiction of a machinelearning model and parse that graphical depiction into instructions forcreating and training a corresponding neural network. Machine learningframework 238 can be configured to select model training parameters.This selection can be based on model performance feedback received fromanother component of machine learning framework 238. Machine learningframework 238 can be configured to provide trained models anddescriptive information concerning the trained models to model memory234.

Data assignment program 240 may be configured to select data from dataset 232 and create training sets and validation sets, as describedherein. Data assignment program 240 may include a random numbergenerator or a pseudo-random number generator to permit randomassignment of individual data points to training sets or validationsets. Data assignment program 240 may be configured to accessinformation stored in the data set 232, and also to write additionalinformation reflecting assignment to training sets and validation setsto the data set 232. Data assignment program 240 may be configured toread and write data using SQL PARQUET, AVRO, SQLITE, POSTGRESQL, MYSQL,ORACLE, HADOOP, CSV, JSON, PDF, JPG, BMP, and/or other data formats.

FIG. 3 depicts an exemplary system for generating a survey datasetsuitable for use with process 100, consistent with disclosedembodiments. As depicted in FIG. 3 , users 302 may be associated withtransactions (e.g., users 302 may conduct the transactions, beassociated with a financial services account used to conduct thetransactions, be legally responsible for the transactions, or the like).Server 304 can be configured to record transaction data detailing thetransactions. In some embodiments, server 304 can be part of a financialservice system associated with a financial institution or can obtain thetransaction data from such a financial service system. The transactiondata can include, as a non-limiting example, an indication a userassociated with the transaction (e.g., account number, user identifier,or the like), transaction amount, transaction time, geographictransaction location, counterparty (e.g., the individual or entity withwhom the transaction occurred), time since prior transaction or priorsimilar transaction (e.g., when the transaction is a car purchase, timesince car previously purchased), transaction method, or the like).Server 304 can include one or more computing devices configured toimplement process 100 (e.g., computing system 200). In this non-limitingexample, the transaction data comprises independent variables asdescribed above with regards to process 100.

Surveys may be sent to users 302 in response to performance of thetransactions and consistent with disclosed embodiments. The surveys maybe automatically provided by a computing system (e.g., server 304 oranother system). The provided surveys may include written surveys (e.g.,email, text message, mail, or the like) and/or telephonic surveys (e.g.manual calling, interactive voice response, or the like). The particularmethod by which surveys are provided is not intended to be limiting. Asurvey may include questions concerning the transaction (e.g., the easeof the transaction, the behavior of the counterparty to the transaction,the terms of the transaction, the perceived benefits of the transaction,the relative value of the transaction compared to other potentialtransactions, or the like). A survey may also include questionsconcerning future user behavior (e.g., likelihood that the user willrepeat the transaction, likelihood that the user will recommend thetransaction to others). The survey may also be configured to enableassociation of the survey response data with corresponding transactiondata. For example, the survey can include a user identifier or atransaction identifier.

Survey response data can be obtained by a computer system (e.g., server304 or another system). For example, electronic survey response data canbe received directly by the computer system, while other forms of surveyresponse data can be input (e.g., scanned or manually entered) to thecomputer system. The particular method by which survey response data isreceived is not intended to be limiting. In some embodiments, additionalprocessing can be performed on survey response data. For example,completed surveys can be analyzed to determine a net user value. In thisexample, a survey may include a question asking users how likely theyare to recommend a colleague to use a certain service on a scale of 0 to10. Additional processing may be performed to assign a category of“active promoter” to users who respond with a 9 or 10, “passivepromoter” to users responding with 7 or 8, and “detractor” to usersresponding with scores below 7. In some embodiments, the category,rather than the numerical score, may then be used as a dependentvariable. In various embodiments, a percentage of detractors can besubtracted from the percentage of promoters to get the net user value.The net user value can then be used as a dependent variable. In thisnon-limiting example, survey response data (or values derived from thesurvey response data during additional processing) comprise a dependentvariable as described above with regards to process 100.

In some embodiments, server 304 can be configured to associate thesurvey response data with corresponding transaction data, therebycreating dataset 306. With reference to FIG. 3 , transaction data fortransactions associated with users A, B, C, and D can be associated withsurvey response data provided by user A, B, C, and D. In this toyexample, data set 306 contains four data points, one for each of User A,B, C, and D. In some aspects, each data point corresponds to atransaction and includes the (independent) transaction data and the(dependent) survey response data for the transaction. FIG. 3 illustratesa small data set for simplicity. In practice, data sets would likely befar larger and more complex. For example, a data set may include manymore independent variables, sources (e.g., users), and dependentvariables. Additionally, a data set may be missing data. For example,users may not respond to surveys, or to all questions in a survey,resulting in independent data that are uncorrelated to dependent data.In some embodiments, the data set may omit incomplete or uncorrelateddata, or indicate missing data using a Not-A-Number symbol, or the like.As would be appreciated by one of skill in the art, in some applicationsonly a small fraction of users that conduct transactions will providesurvey response data.

FIG. 4 depicts exemplary training machine learning models, consistentwith disclosed embodiments. In the non-limiting example depicted in FIG.4 , data set 400 contains individual data points from five sources 402:A, B, C, D, and E (corresponding to the users depicted in FIG. 3 ). Eachindividual data point contains (independent) transaction data comprisingvalues 404 for three variables. Additionally, each individual data pointcontains (dependent) survey data comprising a single Boolean-valuedvariable. The number and data type of the independent and dependentvariables depicted in this example are not intended to be limiting.

As depicted in FIG. 4 , the at least one computing system of process 100can be configured to generate three training set/validation set pairsfrom dataset 400. Pair 408 includes training set 1, containingindividual data points A, B, and C. Validation set 1 contains individualdata points D and E. Similarly, pair 410 includes training set 2,containing data points C, D, and E, and validation set 2, containingdata points A and B. Pair 412 includes training set 3, containing datapoints B, C, and D, and validation set 3, containing data points A andE. In this example, no data point is omitted from a paired training setand validation set, and no data point is repeated between a training setand a validation set in a pair. For instance, in pair 408, each of theindividual data points are either in training set 1 or validation set 1.Furthermore, data point A is only in training set 1 and not invalidation set 1. Pair 412 also shows that data points do not need to besequentially assigned. For instance, validation set 3 includes datapoints A and E. Each of the pairs 408, 410, and 412 may then be used totrain models and calculate accuracy metric values.

FIG. 5 depicts an exemplary method for identifying predictive machinelearning parameters using multiple machine learning models, consistentwith disclosed embodiments. In the non-limiting example depicted in FIG.5 , the at least one computer of process 100 can be configured togenerate three models based on three collections of input data (inputdata 502, input data 504, and input data 506). These three collectionsof input data can each comprise a training dataset and a validationdataset. In some embodiments, a seed may be associated with eachcollection of input data. In some embodiments, this seed may have beenused to generate the training and validation datasets and/or may be usedto train the model (e.g., by initializing a random number generator usedto generate the training and validation datasets and/or used to trainthe model). A model can be created for each set of input data (models508, 510, and 512). For example, each model may comprise weightsassociated with each independent variable, as illustrated in FIG. 5 .The at least one computing system of process 100 can then be used tocalculate an accuracy metric for each model.

In FIG. 5 , for instance, input 502 is used to create model 1. Model 1is then evaluated using the validation set from input 502, yielding anaccuracy metric value. In this non-limiting example, the at least onecomputing system can calculate an AUC (e.g., AUC 1, AUC 2, and AUC 3)for each model. As can be appreciated, other accuracy metrics may bechosen—this example is not intended to be limiting. The at least onecomputing system of FIG. 100 can be configured to analyze the calculatedAUC values (AUC analysis 514). This analysis may involve determiningstatistics of the calculated AUC values. For example, a distribution ofAUC values can be estimated. This estimated distribution can be analyzedto determine whether an accuracy criterion for the machine learningmodels has been satisfied. In this non-limiting example, the at leastone computing system can be configured to determine an average AUCvalue, which can be compared to a predetermined threshold value.Alternatively, the accuracy criterion can depend on one or more of thestandard deviation, mean, median, mode, or other statistic of theestimated distribution.

If the accuracy metric is below the threshold value, indicating that themodels are unable to produce reliable predictions, the process may stop.On the other hand, if the accuracy metric is above the threshold value,ranks 516 are determined for the independent variables. For example, asshown in the ranks 516, independent variable 3, corresponding to weight3, is ranked first because it has the strongest correlation to thedependent variable. The correlation strength may be determined, forinstance, by calculating the average weight for each independentvariable across all models, or by summing the weights associated witheach independent variable across all models. As will be appreciated byone skilled in the art, other methods to rank independent variables maybe employed.

Systems and methods disclosed herein involve unconventional improvementsover prior methods of training and using machine learning models.Descriptions of the disclosed embodiments are not exhaustive and are notlimited to the precise forms or embodiments disclosed. Modifications andadaptations of the embodiments will be apparent from consideration ofthe specification and practice of the disclosed embodiments.Additionally, the disclosed embodiments are not limited to the examplesdiscussed herein.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations of theembodiments will be apparent from consideration of the specification andpractice of the disclosed embodiments. For example, the describedimplementations include software, but methods consistent with thepresent disclosure may be implemented as hardware alone.

Computer programs based on the written description and methods of thisspecification are within the skill of a software developer. The variousfunctions, scripts, programs, or modules can be created using a varietyof programming techniques. For example, programs, scripts, functions,program sections or program modules can be designed in or by means oflanguages, including Javascript, C, C++, Java, PHP, Python, Ruby, Perl,or other programming or scripting languages. One or more of suchsoftware sections or modules can be integrated into a computer system,non-transitory computer-readable media, or existing communicationssoftware. The programs, modules, or code can also be implemented orreplicated as firmware or circuit logic.

Moreover, while illustrative embodiments have been described herein, thescope includes any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations or alterations based on the presentdisclosure. The elements in the claims are to be interpreted broadlybased on the language employed in the claims and not limited to examplesdescribed in the present specification or during the prosecution of theapplication, which examples are to be construed as non-exclusive.Further, the steps of the disclosed methods can be modified in anymanner, including by reordering steps or inserting or deleting steps. Itis intended, therefore, that the specification and examples beconsidered as exemplary only, with a true scope and spirit beingindicated by the following claims and their full scope of equivalents.

1-20. (canceled)
 21. A method comprising: collecting independent datarepresenting independent variables; collecting dependent datarepresenting a dependent variable; correlating the independent data withthe dependent data; creating a data set comprising the correlatedindependent and dependent data; creating a plurality of training setsand a plurality of validation sets from the data set; associating eachtraining set with a single validation set; and training a neural networka plurality of times with the training sets to create a plurality ofmodels.
 22. The method of claim 21, wherein: the dependent variablecomprises a user satisfaction metric; and the independent variablescomprise business transaction variables.
 23. The method of claim 22,wherein: the dependent data comprises data generated from surveys; andthe independent data comprises data generated by analysis of businessrecords.
 24. The method of claim 23, wherein: the dependent datacomprises a net value calculated by analysis of survey results.
 25. Themethod of claim 23, wherein the business records comprise at least oneof: sales value; sales quantity; sales frequency; transaction time; userreferrals; or a success ratio, calculated as a ratio of completedtransactions to initiated transactions.
 26. The method of claim 21,further comprising: generating a plurality of unique seeds; and trainingthe neural network the plurality of times with the seeds to create theplurality of models, wherein the unique seeds are generated by one ofrandom or pseudo-random procedures.
 27. The method of claim 21, furthercomprising: calculating accuracy metric values for the models using thevalidation sets associated with the training sets used to createrespective models; performing a statistical analysis of the accuracymetric values; and ranking the independent variables by a strength ofcorrelation of individual independent variables with the dependentvariable, when a metric of the statistical analysis exceeds a threshold,wherein: the accuracy metric comprises an Area Under the Curve (AUC);the statistical analysis comprises calculating an average; and themetric of the statistical analysis comprises the average.
 28. The methodof claim 21, wherein associating each training set with a singlevalidation set further comprises: pairing training and validation setssuch that no individual data point is in both a training set and avalidation set of a pair.
 29. The method of claim 28, wherein: a numberof training sets and a number of validation sets is equal to a number ofunique seeds; a number of the models is equal to the number of uniqueseeds; and each model is created by training with one seed and one pairof a training set and a validation set.
 30. The method of claim 28,wherein: individual data points are assigned to a training set or avalidation set based on one of a random number or a pseudo-randomnumber.
 31. A non-transitory computer-readable medium containinginstructions to perform operations comprising: collecting independentdata representing independent variables; collecting dependent datarepresenting a dependent variable; correlating the independent data withthe dependent data; creating a data set comprising the correlatedindependent and dependent data; creating a plurality of training setsand a plurality of validation sets from the data set; associating eachtraining set with a single validation set; and training a neural networka plurality of times with the training sets to create a plurality ofmodels.
 32. The medium of claim 31, wherein: the dependent variablecomprises a user satisfaction metric; and the independent variablescomprise business transaction variables.
 33. The medium of claim 32,wherein: the dependent data comprises data generated from surveys; andthe independent data comprises data generated by analysis of businessrecords.
 34. The medium of claim 33, wherein: the dependent datacomprises a net value calculated by analysis of survey results.
 35. Themedium of claim 33, wherein the business records comprise at least oneof: sales value; sales quantity; sales frequency; transaction time; userreferrals; or a success ratio, calculated as a ratio of completedtransactions to initiated transactions.
 36. The medium of claim 31,wherein the operations further comprises: generating a plurality ofunique seeds; and training the neural network the plurality of timeswith the seeds to create the plurality of models, the unique seeds aregenerated by one of random or pseudo-random procedures.
 37. The mediumof claim 31, wherein the operations further comprises: calculatingaccuracy metric values for the models using the validation setsassociated with the training sets used to create respective models;performing a statistical analysis of the accuracy metric values; andranking the independent variables by a strength of correlation ofindividual independent variables with the dependent variable, when ametric of the statistical analysis exceeds a threshold, wherein theaccuracy metric comprises an Area Under the Curve (AUC); the statisticalanalysis comprises calculating an average; and the metric of thestatistical analysis comprises the average.
 38. The medium of claim 31,wherein associating each training set with a single validation setfurther comprises: pairing training and validation sets such that noindividual data point is in both a training set and a validation set ofa pair.
 39. The medium of claim 38, wherein: a number of training setsand a number of validation sets is equal to a number of unique seeds; anumber of the models is equal to the number of unique seeds; and eachmodel is created by training with one seed and one pair of a trainingset and a validation set.
 40. The medium of claim 38, wherein:individual data points are assigned to a training set or a validationset based on one of a random number or a pseudo-random number.